Reference clustering

This document describes the process we used to obtain a reference clustering from anti-virus labels to evaluate the malware clustering techniques presented in [3]. This text was included in an early version of [3], but it had to be discarded because of space limitations. Since this reference dataset has been made available and is being used by several researchers, it seems useful to provide detailed information on how it was obtained. 1 Reference Clustering When an anti-virus programs recognizes a malicious programs, it assigns a name to this binary. Typically, virus labels are hierarchically structured. That is, each label has at least one part denoting the malware family and one part describing the particular variant. However, there is no standard naming convention. Each anti-virus vendor creates and assigns its own virus labels. This is why the same file is typically labeled differently by different vendors. Moreover, the granularity of virus labels varies between virus-scanners. Mcafee and Grisoft, for example, have very general labels, where a single name covers a broad range of malware instances. Mcafee also assigns labels such as ‘Generic BackDoor‘ that have neither a variant nor a family part. Kaspersky, on the other hand, consistently uses labels such as ‘EmailWorm.Win32.Zhelatin.jz‘, which allows for the straight forward extraction of a family and variant name. It is possible to cluster a given set of malware samples based on the labels produced by an anti-virus program. However, as pointed out in [2], virus scanners are not particularly well-suited for clustering a given set. First, they